Recently there have been an increasing number of depth sensors that are available at relatively low prices. In an example, a sensor unit that communicates with a video game console includes a depth sensor. In another example, computing devices (desktops, laptops, tablet computing devices) are being manufactured with depth sensors therein. A sensor unit that includes both a color camera as well as a depth sensor can be referred to herein as a depth camera. Depth cameras have created a significant amount of interest in applications such as three-dimensional shape scanning, foreground-background segmentation, facial expression tracking, amongst others.
Depth cameras generate simultaneous streams of color images and depth images. To facilitate the applications discussed above (and other applications that employ color images and depth images), the depth sensor and color camera may be desirably calibrated. More specifically, both the color camera and the depth sensor have their own respective coordinate systems, and how such coordinate systems are aligned with respect to one another may be desirably determined to allow pixels in a color image generated by the color camera to be effectively mapped to pixels in a depth image generated by the depth sensor and vice versa.
Many difficulties exist with respect to calibrating a color camera and depth sensor. For example, color cameras have been calibrated utilizing colored patterns. Colored patterns, however, cannot be analyzed in a depth image, as such image does not include captured colors (e.g., corners of a pattern are often indistinguishable from other surface points in a depth image). Furthermore, although depth discontinuity can be observed in a depth image, boundary points of an object are generally unreliable due to unknown depth reconstruction mechanisms utilized in the depth sensor.
An exemplary approach to calibrate a color camera and depth sensor is to co-center an infrared image with a depth image. This may require, however, external infrared illumination. Additionally, commodity depth cameras typically produce relatively noisy depth images, rendering it difficult to calibrate the depth sensor with the color camera.